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Section  1:  Introduction 


Applied  Science  Associates  recently  evaluated  the  F- 16  Simulated 
Aircraft  Maintenance  Trainers  (SAMTs).  The  study  included  both 
training  effectiveness  and  cost  effectiveness,  and  a  formal  and 
informal  evaluation  of  two  instructional  features  of  the  SAMTs.  A 
representative  subset  of  the  SAMTs  was  studied.  Students  and 
instructors  shared  their  opinions  and  experiences,  and  student  task 
performance  data  were  collected  and  analyzed. 

Although  the  information  gathered  in  the  study  was  not  always 
conclusive,  many  interesting  lessons  were  learned  about  how  to  design 
future  maintenance  trainers,  as  well  as  how  to  conduct  training  and 
cost  effectiveness  studies.  The  purpose  of  this  paper  is  to  present 
these  lessons.  Section  2  presents  background  information  on  the 
trainers  and  the  F- 16  training  program;  Section  3  discusses  the  cost 
effectiveness  of  maintenance  trainers  in  general  and  of  the  SAMTs  in 
particular;  and  Sections  4  to  6  present  specific  lessons  learned  in  the 
areas  of  trainer  design,  trainer  evaluation,  and  training  systems.  The 
lessons  learned  are  written  such  that  each  section  can  stand  alone;  the 
paper  need  not  be  read  cover  to  cover;  i.e.,  this  paper  is  designed  to 
be  a  reference  document  for  System  Program  Office  (SPO)  acquisition 
managers.  An  index  is  provided  at  the  end  to  facilitate  use  of  the 
paper  as  a  reference  document. 

This  paper  is  a  summary  of  the  F- 16  SAMT  study.  Study  results 
will  be  documented  elsewhere  in  future  papers  and  reports. 


Section  2:  Background  Information 


The  F- 16  SAMTs  are  used  in  Field  Training  Detachment  (FTD) 
courses.  Students  in  these  courses  are  either  3-levels  directly  out  of 
technical  school  (sometimes  after  other  assignments)  or  5-  to  7-levels 
who  have  experience  either  on  other  aircraft  or  other  systems.  The 
courses  are  not  self-paced  and  the  instructor  is  present  at  all  times. 
After  the  FTD  course,  students  participate  in  On-the-Job  Training 
(OJT) . 

Five  trainers  used  in  four  different  FTD  courses  were  studied. 

One  of  the  trainers  was  configured  as  a  cockpit  and  the  others  were 
typical  flat  panel  trainers.  These  trainers  were  selected  from  the  set 
of  12  because  they  could  be  made  available  for  the  study,  and  they 
represented  the  two  types  of  trainer  configurations  found  in  the  set. 
Each  trainer  consists  of  a  master  console  (containing  the  computer  and 
the  instructor  controls)  and  panels  [or,  in  the  case  of  the  Engine  Run 
trainer,  a  three-dimensional  (3-D)  mockup]  representing  the  simulated 
system.  Each  trainer  also  has  a  built-in  slide  projector.  The  courses 
in  which  the  trainers  are  used  have  an  average  of  five  students  per 
class  and  range  in  length  from  3  days  to  several  weeks.  The  following 
trainers  were  evaluated: 

TFE-2  Avionics  Flight  Controls 
TFE-4  Electrical  Systems 
TFE-10  Engine  Start 
TFE-11  Engine  Diagnostics 
TFE-12  Engine  Run 

The  TFE-10  and  TFE-11  are  used  in  the  same  course,  and  for  most 
purposes  will  be  considered  together.  The  Pneudraulics  trainer  was 
also  initially  targeted  for  study  but  was  not  included  due  to  the 
decision  to  move  the  trainer  to  resident  school. 

As  stated  previously,  the  complete  results  of  the  evaluation  will 
not  be  discussed  in  this  paper.  However,  it  should  be  mentioned  that 
overall  the  results  of  the  study  speak  well  for  the  effectiveness  of 
simulation  in  maintenance  training.  High  percentages  of  students 
passed  all  of  the  tasks  investigated  and  passed  by  standards  more 
exacting  than  those  adopted  by  the  FTD.  That  is,  the  proficiency 
needed  to  pass  a  task  in  the  evaluation  context  was  often  greater  than 
that  needed  to  pass  the  course.  Also,  the  attitudes  of  both  students 
and  instructors  were  generally  positive  toward  the  trainers.  For 
example,  the  instructors  generally  agreed  that  using  the  SAMTs  made 
teaching  less  difficult  and  improved  the  quality  of  instruction. 
Specific  negative  responses  and  ineffective  features  mentioned  later  in 
this  paper  should  be  balanced  against  these  generally  positive 
findings. 
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Section  3:  Trainer  Cost  Effectiveness 


The  cost  effectiveness  portion  of  the  F- 16  SAMT  study  had  several 
goals.  The  first  was  to  determine  the  total  life  cycle  costs  for  each 
selected  SAMT.  That  is,  it  was  planned  to  calculate  not  only  the 
acquisition  cost,  but  also  operation  and  maintenance  costs.  These 
calculations  were  to  include  the  costs  of  the  trainers'  instructional 
features.  Another  goal  of  the  cost  study  was  to  estimate  the  costs  of 
equivalent  Actual  Equipment  Trainers  (AETs).  That  is,  the  study  was 
intended  to  determine  acquisition,  operation,  and  maintenance  costs  for 
hypothetical  AETs  that  could  be  used  to  meet  the  same  training 
objectives  as  did  the  F- 16  SAMTs.  These  estimated  AET  costs  were  to  be 
compared  with  the  determined  costs  for  the  trainers. 

Unfortunately,  these  goals  proved  to  be  difficult  to  meet.  The 
cost  data  available  to  the  study  were  extremely  limited.  Most  of  the 
data  made  available  to  the  study  were  received  toward  the  end  of  the 
study,  so  that  only  limited  analysis  was  possible;  however,  some 
conclusions  could  be  drawn.  The  methodology  and  findings  of  the  cost 
effectiveness  portion  of  the  study  are  briefly  described  in  the 
following  sections. 


3.1  Life  Cycle  Costs 

The  cost  model  used  in  the  study  is  not  described  in  detail  here, 
as  it  is  adequately  documented  elsewhere.  However,  the  concept  of  life 
cycle  costs  must  be  understood.  Rather  than  comparing  trainers  solely 
on  the  basis  of  acquisition  costs,  it  is  important  that  costs  over  the 
entire  life  of  the  trainer  be  considered.  That  is,  one  trainer  that 
initially  costs  more  than  another  might  be  less  costly  to  operate  and 
maintain,  and  so  prove  to  be  less  costly  overall.  In  the  SAMT  study, 
acquisition,  operation,  and  maintenance  costs  were  considered  for:  (a) 
facility,  (b)  equipment,  (c)  materials,  (d)  personnel,  (e)  supply, 
and  (f)  miscellaneous. 

Data  were  not  available  for  all  factors;  however,  when  possible,  each 
area  will  be  addressed  in  the  following  sections. 


3.2  SAMT  Acquisition  Costs 

The  acquisition  costs  for  the  five  SAMTs  under  study  are  reported 
in  Table  1.  These  costs  were  derived  from  a  list  of  prices  of  SAMTs 
delivered  to  several  different  sites.  The  costs  reported  in  Table  1 
are  average  (mean)  unit  costs.  The  average  costs  were  computed  by 
information  provided  by  the  F- 16  System  Program  Office  ( SPO) .  The 
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costs  reported  to  the  F- 16  SPO  by  equipment  prime  contractors  were  not 
consistent,  making  it  difficult  to  compute  a  meaningful  average.  For 
example,  some  unit  costs  included  administrative  and  some  included 
research  and  development  (R&D)  costs.  Unfortunately,  these  costs  were 
not  listed  as  line  item  costs;  thus,  they  could  not  be  separated  from 
the  total  unit  price  in  a  precise  way.  The  average  unit  costs  reported 
in  Table  1  were  computed  by  averaging  the  price  of  units  which  did  not 
include  any  additional  charges  (administrative  and  R&D  costs).  The 
vendor  administrative  and  R&D  costs  listed  in  Table  1  were  derived  by 
subtracting  the  average  unit  costs  from  all  those  unit  costs  where 
these  additional  costs  were  included.  All  the  unit  costs  reported  in 
Table  1  include  delivery  charges,  which  could  not  be  separated  from  the 
total  unit  costs  because  of  the  nature  of  the  data  provided  to  the  F - 1 6 
SPO. 

As  Table  1  clearly  shows,  in  all  cases  the  R&D  costs  were  much 
greater  than  the  unit  price  of  the  trainers.  R&D  costs  (and  the  listed 
vendor  administrative  costs)  are  one-time  costs.  If  man>  trainers  are 
to  be  acquired,  a  relatively  low  unit  price  will  outweigh  high, 
one-time  costs.  For  a  comparison  with  expected  acquisition  costs  for 
an  AET,  see  Section  3.3.4. 


Table  1.  SAMT  Acquisition  Costs 


Avionics 

Electrical 

Engine  Start 

Engine 

Diagnostics 


Average  Price 
(Including 
Delivery) 


Approximate 
Vendor  Adminis¬ 
trative  Costs 


Approximate 
R&D  Costs 


$357,565  $170,338  $2,137,807 

261,332  75,932  1,251,783 

175,024  105,208  1,132,174 


333,431 


209,926 


2,294,180 


In  the  cost  study,  each  of  the  five  trainers  was  compared  with  a 
hypothetical  AET.  This  involved  generating  hypothetical  designs  for 
the  AET  that  could  be  used  to  meet  the  SAMT's  training  objectives.  In 
this  paper,  the  findings  for  the  Engine  Run  course  will  be  emphasized. 
The  next  section  (Section  3.3.1)  briefly  discusses  comparative  costs 
per  student  hour  for  all  the  courses;  the  following  sections  discuss 
the  hypothetical  Engine  Run  AET,  and  how  it  compared  with  the  Engine 
Run  SAMT. 


3.3.1  Relative  Costs  Per  Student 

A  small  amount  of  cost  data  was  available  to  the  study,  asid  rorr 
the  acquisition  data.  Although  the  data  were  limited,  estimates  i 
relative  costs  per  student  and  per  student  hour  could  be  generatec 
These  estimates  for  the  SAMTs  and  AETs  in  all  four  courses  are 
presented  in  Table  2. 

Only  those  operation  and  maintenance  costs  which  could  be 
estimated  for  both  the  SAMTs  and  the  hypothetical  AETs  were  included. 
These  costs  consisted  of  the  following:  facility  maintenance  costs, 
instructor  salaries,  vendor  maintenance  costs,  and  fuel  costs  for 
supplementary  training.  The  largest  single  factor  was  the  cost  of 
fuel,  which  is  incurred  for  AET  training,  but  not  for  SAMT  training. 

It  was  logically  determined,  by  analyzing  the  hypothetical  AET  designs, 
that  facility  maintenance  costs  would  be  higher  for  the  AET,  and 
instructor  salaries  would  be  equal  for  the  two  training  systems. 

Vendor  maintenance  is  an  additional  cost  incurred  by  SAMTs  but  not  by 
AETs,  since  an  AET  is  typically  maintained  by  Air  Force  personnel 
rather  than  vendors.  It  is  worth  noting  that  costs  per  student  and  per 
student  hour  are  higher  for  the  AETs  in  all  cases. 

These  factors,  as  they  apply  to  the  Engine  Run  course,  are 
described  in  more  detail  in  the  following  sections. 


3.3.2  AET  Oesign  for  Engine  Run 

In  order  to  estimate  costs  associated  with  the  hypothetical  AET, 
it  was  necessary  to  generate  a  hypothetical  design.  Obviously,  an  AET 
could  take  many  different  forms  which  might  vary  widely  in  cost.  Based 
on  the  training  requirements  of  the  Engine  Run  course,  a  single 
possible  design  was  chosen  and  was  used  as  the  basis  for  cost 
estimates. 


The  hypothetical  design  for  the  Engine  Run  AET  consists  of  an 
actual  F- 16  cockpit  with  a  full  set  of  controls  and  displays.  Those 
components  which  are  powered  by  the  aircraft  battery  would  be  fully 
operational,  while  those  dependent  on  the  engine  would  not  be 
functional.  That  is,  the  AET  would  have  electrical  power  equivalent  to 
that  of  the  aircraft,  but  would  not  contain  or  simulate  an  engine. 

The  hypothetical  design  has  various  implications  for  acquisition, 
operation,  and  maintenance  costs.  The  expected  costs  for  this  AET  are 
contrasted  with  the  Engine  Run  SAMT  costs  in  the  following  sections. 

The  design  also  has  implications  for  training,  as  will  be  discussed  in 
Section  3.4. 


3.3.3  Engine  Run  Life  Cycle  Costs 

The  known  life  cycle  costs  for  the  Engine  Run  SAMT  and  corres¬ 
ponding  hypothetical  AET  are  reported  in  Table  3.  The  costs  listed  in 
Table  3  do  not  represent  all  of  the  costs  associated  with  either  type 
of  trainer.  However,  several  of  the  most  important  cost  factors  (such 
as  fuel  costs)  are  included.  It  is  believed  that  the  costs  included  in 
the  table  are  indicative  of  the  total  relative  costs.  That  is, 
although  the  totals  listed  are  less  than  actual  expenses,  they  show  the 
areas  where  the  costs  of  the  AET  would  be  greater  or  less  than  the 
costs  of  the  SAMT.  The  derivation  of  the  estimated  costs  is  presented 
in  the  following  sections. 


3.3.4  Acquisition  Comparison 

The  unit  price  of  the  hypothetical  AET  would  be  greater  than  the 
unit  cost  for  the  SAMT.  The  SAMT,  however,  would  have  a  much  greater 
R&D  cost.  As  shown  in  Table  3,  the  cost  for  the  first  unit  (including 
R&D)  would  be  much  higher  for  the  SAMT.  Over  the  first  four  units, 
however,  the  costs  would  balance  out.  For  the  next  four  units,  the 
SAMT  per  unit  cost  would  be  much  lower.  Thus,  the  cost  effectiveness 
of  the  trainers  would  depend  in  part  on  how  many  units  were  purchased. 
In  this  case,  if  more  than  four  units  were  purchased,  the  SAMTs  would 
be  less  expensive  overall,  despite  higher  initial  costs.  Although 
these  particular  numbers  are  specific  to  the  Engine  Run  SAMT  and 
hypothetical  AET,  this  pattern  would  be  expected  to  apply  to  other 
cases.  That  is,  a  simulator  would  usually  have  high  R&D  costs,  but  low 
per-unit  costs;  thus,  in  the  long  run  simulators  will  tend  to  cost  less 
than  AETs. 

The  unit  costs,  vendor  administrative  costs,  and  R&D  costs  for  the 
SAMTs  were  derived  from  cost  data  provided  by  the  F - 16  SPO,  as 
explained  in  Section  3.2.  The  unit  cost  of  the  AET  was  harder  to 
estimate. 
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Table  3.  '■'nown  Life  Cycle  Costs 
Engine  Run  SAMT  vs.  Hypothetical  AET 


Hypothetical 


Acquisition 

SAWT 

AET 

-  Unit  Price 

$  195,035 

$  500,000 

-  Vendor  Administrative  Costs 

206,067 

206,067 

-  R&O 

1,268,091 

100,000 

Total  Acquisition  Cost, 

First  Unit: 

1,669,193 

806,067 

First  Four  Units: 

2,254,798 

2,306,067 

Second  Four  Units: 

780,140 

2,000,000 

Operation  and  Maintenance 


-  Instructor  Salaries 

35,120 

35,120 

-  Fuel  Cost 

0 

437,500 

-  Vendor  Maintenance 

1,400 

0 

-  Air  Force  Maintenance 

-- 

-- 

-  Spare  Parts 

10,667 

27,778 

Total  Yearly  Operation 

and  Maintenance  Cost: 

47,187 

500,398 

Modification 

-  Update  Costs 

390,070 

50,000 

Total  Estimated  Cost, 

First  Year,  First  Unit: 

2,106,450 

1,356,465 

Second  Unit: 

630,292 

1,050,398 

Several  F- 16  trainers  utilizing  some  actual  equipment  exist,  such  as  a 
canopy  system  trainer,  a  cockpit  procedures  trainer,  and  an  egress 
trainer.  The  prices  of  these  trainers  ranged  from  $375,000  to 
$600,000.  The  figure  $500,000  was  chosen  as  a  representative  figure. 
Other  AETs  have  cost  as  much  as  $700,000  to  $1,000,000.  Thus,  $500,000 
is  fairly  conservative,  although  not  the  lowest  possible  figure. 

It  was  arbitrarily  assumed  that  costs  for  design  reviews  would  be 
approximately  the  same  for  the  AET  and  the  SAMT.  The  figure  for  R&D 
for  the  AET  was  derived  from  data  on  a  F- 16  canopy  system  AET.  For 
that  trainer,  the  price  of  the  first  unit  was  $100,000  higher  than  for 
later  units.  It  is  assumed  that  this  difference  represents  R&D  costs. 
(That  is,  research  and  development  associated  with  determining  what 
components  of  actual  equipment  to  use,  how  they  are  to  be  modified, 
etc.)  This  figure,  although  arbitrary,  seemed  to  be  a  reasonable 
estimate  for  the  Engine  Run  AET  R&D  cost. 


3.3.5  Operation  and  Maintenance  Comparison 


Many  factors  influence  operation  and  maintenance  costs.  Data  were 
available  for  only  a  few  of  these  factors.  Instructor  salaries  were 
estimated  (based  on  a  1981  wage  rate)  for  two  full-time  instructors 
(for  1  year)  in  both  the  SAMT  and  the  AET  courses.  Fuel  cost  for  AET 
supplementary  training  was  estimated  by  calculating  the  pounds  of  fuel 
used  in  an  average  engine  run  (1,750),  times  the  number  of  students 
yearly,  at  a  rate  of  $1.25  per  pound.  Note  that  this  is  a  conservative 
estimate,  since  more  than  one  practice  run  per  student  might  be 
necessary. 

The  calculations  for  maintenance  costs  do  not  include  maintainer 
salaries,  since,  for  both  systems,  maintenance  is  performed  by 
instructors.  The  estimated  cost  for  SAMT  vendor-provided  maintenance 
was  derived  from  figures  provided  by  personnel  at  Hill  AFB  and  McDill 
AFB.  Similarly,  data  on  the  number  and  kind  of  spare  parts  needed 
yearly  for  the  SAMT  were  provided,  from  which  a  cost  figure  was 
derived.  It  was  difficult  to  obtain  such  data  for  the  AET.  The  figure 
listed  in  Table  3  for  the  AET  was  obtained  by  assuming  that  the 
relative  cost  of  units  and  spare  parts  would  be  the  same  for  SAMT  and 
AET.  That  is,  spare  parts  for  the  SAMT  were  known  to  cost  about  18 
percent  of  the  unit  price.  Thus,  the  figure  listed  for  AET  spare  parts 
represents  18  percent  of  the  unit  price.  Note  that  the  SAMT  figure  was 
not  for  an  entire  spares  package  but  for  parts  actually  used. 


3.3.6  Modification  Comparison 

The  costs  associated  with  modifying  the  SAMT  (especially  SAMT 
software)  is  known  to  be  high.  Specific  cost  figures  were  not 
available,  but  informal  estimates  made  by  Air  Force  personnel  ran  as 
high  as  2  1/2  times  the  initial  acquisition  cost.  Vendor  modifications 
are  necessary  to  keep  the  trainers  up  to  date  and  useful  because  the 
SAMT  does  not  have  local  update  capabilities.  (See  Section  4.4.) 

For  SAMT  updates,  the  cost  figure  listed  in  Table  3  is  twice  the 
unit  price.  The  cost  listed  for  AET  updates  is  10  percent  of  the 
acquisition  cost.  This  estimate  was  based  on  the  number  and  kind  of 
Engineering  Change  Proposals  (ECPs)  that  the  F- 16  aircraft  has  in  an 
average  year.  The  estimated  total  yearly  costs  for  the  first  unit  and 
for  the  second  and  all  following  units  include  yearly  update  costs,  as 
well  as  acquisition  and  yearly  operation  and  maintenance  costs.  Note 
that  costs  for  the  first  unit,  which  include  R&D  costs,  will  be  higher 
for  the  SAMT,  but  for  later  units,  the  AET  cost  is  higher. 


3.4  Value  for  Training 

The  comparisons  discussed  above  relate  only  to  cost,  not  to 
training  benefits.  However,  it  is  important  to  remember  that  training 
value  will  vary  also.  For  instance,  the  AET  cannot  present  emergency 
tasks  in  the  way  a  simulator  can.  Using  the  AET,  the  instructor  might 
talk-through  emergency  symptoms  and  procedures,  but  the  student  cannot 
actually  practice  recognizing  and  responding  to  malfunctions.  Also, 
as  previously  mentioned,  it  was  assumed  that  AET  students  would 
practice  a  normal  engine  run  on  the  aircraft.  However,  on  the  SAMT, 
students  can  have  many  opportunities  to  practice  the  run,  rather  than 
just  one  chance. 

More  practice  on  the  actual  aircraft  could  be  provided,  at  greater 
cost.  However,  hazardous  tasks  really  cannot  be  practiced  on  actual 
equipment.  Thus,  in  certain  ways  a  simulator  will  probably  always  have 
greater  training  value  than  an  AET.  The  extent  of  this  depends  on  the 
nature  of  the  aircraft  system  and  the  tasks  to  be  trained. 


Section  4:  Trainer  Design  Issues 


This  section  presents  several  lessons  learned,  both  positive  and 
negative,  which  relate  to  design  features  of  the  SAMTs.  These  lessons 
are  categorized  into  the  following  areas:  fidelity  levels, 
instructional  features,  hardware  design,  trainer  maintenance  and 
update,  and  task  selection.  Note  that  these  lessons  learned  do  not 
cover  every  feature  of  the  trainers;  rather,  they  are  issues  that 
happened  to  emerge  in  the  course  of  the  study. 


4. 1  Fidelity  Levels 


The  issue  of  fidelity  surfaced  many  times  in  the  study,  and  the 
results  were  interesting.  The  trainers  in  the  four  courses  under  study 
varied  in  their  fidelity  levels.  While  degree  of  fidelity  is  difficult 
to  measure  precisely,  relative  fidelity  was  estimated  logically  by 
talking  to  instructors  and  students.  Instructors  (n=6)  were  asked 
which  of  the  trainers  had  the  highest  and  lowest  fidelity.  The  trainer 
that  was  considered  most  realistic  was  the  Engine  Run  trainer,  which  is 
a  3-0  representation  of  an  F- 16  cockpit.  Most  of  the  controls  and 
displays  are  functional.  The  panel  trainer  considered  lowest  in 
fidelity  was  the  Avionics  trainer.  The  Engine  and  Electrical  trainers 
were  rated  toward  the  middle  of  the  fidelity  continuum.  In  addition  to 
overall  fidelity,  trainers  were  also  rated  as  to  their  relative 
fidelity  regarding  operational  checks  and  malfunction  simulation. 


4.1.1  General  Findings 

Several  training  effectiveness  measures  were  collected  and 
analyzed.  One  of  these  was  a  self-rating  by  the  students  of  their 
confidence  in  performing  tasks  after  training,  both  on  the  aircraft  and 
on  the  SAMTs.  Students  rated  their  own  confidence,  on  a  scale  of  one 
to  seven,  in  performing  specific  tasks  in  their  subject  field.  That 
is,  students  from  the  Engine  Diagnostics  course  rated  their  confidence 
in  performing  Engine  Diagnostics  tasks;  similarly,  students  from  the 
other  courses  rated  their  own  confidence  for  tasks  related  to  the 
subsystem  they  had  studied.  The  tasks  that  were  rated  fell  into  two 
broad  categories:  fault  isolation  and  operational  checks.  Figure  1 
shows  the  average  confidence  reported  by  students  in  each  course,  for 
each  type  of  task. 

The  information  in  Figure  1  can  be  examined  trainer  by  trainer. 

For  two  of  the  courses,  confidence  in  performance  on  the  aircraft  was 
higher  for  fault  isolation  tasks;  for  the  other  two,  confidence  was 
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higher  for  operational  checks.  This  correlated  with  the  fidelity 
levels  reported  by  instructors.  That  is,  trainers  that  were  reported 
to  have  higher  fidelity  in  operational  checks  than  in  malfunction 
simulation  produced  higher  levels  of  confidence  in  operational  checks; 
and  vice  versa.  For  instance,  the  Engine  Diagnostics  trainer,  which 
reportedly  has  high  fidelity  malfunction  simulation,  produced  much 
higher  levels  of  confidence  for  troubleshooting  tasks  than  for 
operational  tasks. 

The  information  in  Figure  1  can  be  used  to  compare  trainers,  but 
only  if  certain  assumptions  are  made.  The  reader  must  decide  if  these 
assuriptions  are  valid,  in  order  to  decide  whether  to  consider  the  data 
in  this  way.  The  first  assumption  is  that  tasks  of  a  given  type  have 
common  characteristics  and  can  be  compared.  That  is,  an  operational 
check  involves  following  procedures,  locating  aircraft  components, 
interpreting  test  equipment  readings,  and  so  on,  regardless  of  the 
specific  subsystem  involved.  In  that  light,  an  Engine  operational 
check  can  be  compared  to  an  Avionics  system  operational  check.  The 
content  is  different,  but  the  type  of  task  is  the  same.  The  second 
assumption  is  that  the  tasks  chosen  for  comparison  were  of 
approximately  equal  difficulty.  The  researchers  attempted  to  choose 
such  tasks,  with  the  assistance  of  instructors  and  other  Subject  Matter 
Experts  (SMEs).  However,  it  is  still  an  assumption  the  reader  must 
make. 

If  the  reader  chooses  to  consider  the  data  in  this  way,  the 
information  in  Figure  1  can  be  seen  as  comparing  the  confidence  levels 
of  students  taught  on  different  trainers.  These  data  also  correspond 
to  the  reported  fidelity  levels.  That  is,  the  Engine  Run  trainer, 
reported  to  be  the  highest  in  fidelity,  produced  the  highest  confidence 
levels  overall;  the  Avionics  trainer,  reported  to  be  lowest  in 
fidelity,  produced  the  lowest  confidence. 

In  addition  to  the  confidence  ratings,  data  were  also  collected  on 
actual  student  performance.  These  data  were  obtained  by  observing 
actual  task  performance,  and  recording  errors.  An  error  was  defined  as 
omission  or  incorrect  performance  of  a  procedural  step.  The  inability 
to  answer  an  important  question  about  a  step  was  also  considered  an 
error.  Certain  errors  were  flagged  as  "critical,"  specifically  those 
that  could  cause  damage  to  the  aircraft  or  create  a  safety  hazard. 
Common  errors  included:  (a)  the  inability  to  locate  an  aircraft 
component  and  (b)  skipping  a  step  or  sequence  of  steps.  The 
proportion  of  errors  made  by  students  in  each  course,  for  the  two  types 
of  tasks  discussed  above,  are  presented  in  Figure  2. 

Note  that,  as  with  the  confidence  ratings,  scores  on  observed 
tasks  varied  from  course  to  course.  The  information  in  Figure  2  may  be 
considered  trainer  by  trainer,  as  an  indication  of  each  trainer's 


relative  effectiveness  in  teaching  operational  checks  and  fault 
isolation.  If  the  reader  chooses  to  make  the  assumptions  specified 
above,  the  data  in  Figure  2  show  that,  as  with  confidence  ratings,  the 
trainer  with  the  highest  fidelity.  Engine  Run,  produced  the  best  scores 
(fewest  errors).  The  trainer  with  the  lowest  reported  fidelity. 
Avionics,  produced  the  most  errors. 

Instructor  attitudes  followed  this  pattern  as  well,  with  the 
Engine  Run  trainer  being  very  well  liked  and  the  Avionics  trainer  liked 
much  less  than  any  of  the  others.  In  general,  instructors  seemed  to 
feel  that  their  trainers  would  be  improved  by  increasing  fidelity, 
especially  in  terms  of  more  realistic  malfunctions  and  more  varied 
troubleshooting  capabilities.  (For  more  details  on  instructional 
features,  see  Section  4.2.) 

It  should  be  noted  that  the  SAMTs  are  relatively  low  fidelity 
simulations  of  an  F- 16  aircraft.  The  preference  for  the  higher  physi¬ 
cal  fidelity  SAMT  does  not  mean  that  fidelity  should  necessarily  be  as 
high  as  is  technically  possible.  It  suggests,  however,  a  limit  on  low 
fidelity.  That  is,  when  fidelity  becomes  too  low,  a  maintenance 
trainer  may  no  longer  be  an  effective  training  medium.  Certain  task 
elements  apparently  are  not  learned  well  on  low  fidelity  devices,  as 
will  be  discussed  in  more  detail  below.  Certainly,  instructor 
acceptance  may  become  a  problem  when  physical  fidelity  is  low. 

As  an  example  of  the  kind  of  extremely  low  fidelity  that  created 
problems,  some  task  segments  and  components  are  represented  on  the 
SAMTs  by  pushbuttons.  The  pushbutton  steps  were  often  forgotten  by 
students,  even  when  the  represented  task  segements  (such  as  "perform 
safe  for  maintenance  check")  were  critical  (i.e.,  would  create  a  safety 
hazard  if  omitted) . 


4.1.2  Teaching  Component  Locations 

In  general,  the  SAMTs  were  very  poor  for  teaching  location  of 
components.  The  design  of  the  trainers  included  graphic 
representations  on  the  panels  and  associated  slides,  which  were 
designed  to  show  students  where  components  are  located  on  the  actual 
aircraft.  However,  the  slides  typically  showed  closeups  of  the 
components,  and  as  such,  could  teach  appearance  but  not  location.  This 
could  be,  in  part,  a  specification  problem.  If  slides  are  intended  to 
teach  location,  it  must  be  clearly  stated  in  the  trainer  specifications 
that  the  slides  should  show  more  than  a  closeup  of  the  component.  If 
slides  and  panel  graphics  (as  typically  used  on  flat  panel  trainers) 
are  not  sufficient  to  teach  location,  as  the  SAMT  data  suggest,  some 
alternative  method  should  be  used  when  component  location  is  an 
important  part  of  a  course.  For  instance,  a  scaled-down  mockup  might 
be  used  along  with  a  panel,  or  actual  aircraft  time  might  be  planned. 
(See  Section  6.2.) 
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4.1.3  Teaching  Difficult  Manipulations 


In  some  cases,  students  repeatedly  made  errors  on  certain  task 
steps  that  involved  components  that  were  simulated  with  fairly  high 
fidelity  (as  judged  by  the  evaluator).  In  these  cases,  the  task  steps 
involved  extremely  sensitive  manipulations  requiring  a  high  degree  of 
skill.  It  seemed  that,  for  difficult  steps  of  this  type,  even  slightly 
reduced  fidelity  can  cause  problems.  For  example,  the  throttle  on  the 
Engine  Start  trainer  was  designed  with  reduced  fidelity,  on  the 
assumption  that  students  would  have  been  previously  exposed  to  the  high 
fidelity  Engine  Run  trainer.  However,  some  of  the  tasks  taught  on  the 
Engine  Start  trainer  required  more  smooth  throttle  manipulations  than 
those  involved  in  a  standard  run.  Perhaps  each  component  on  a  trainer 
should  be  simulated  in  accordance  with  the  most  difficult  task 
involved.  That  is,  fidelity  levels  of  components  on  a  trainer  should 
perhaps  be  determined  by  analyzing  the  most  difficult  task  to  be 
practiced  regularly  on  the  trainer  using  the  component  of  interest. 
Component  fidelity  level  should  not  be  determined  by  finding  the 
average  fidelity  needed  for  all  the  tasks  to  be  trained,  as  is 
typically  done  in  most  design  efforts.  • 


4.1.4  Mode  line 


A  final  point  which  emerged  very  clear-ly  had  to  do  with  the 
trainers'  computational  modeling.  In  general,  the  SAMTs  were  modeled 
on  Technical  Order  (T.O. )  procedures,  rather  than  on  system  data.  The 
exception  was  the  Engine  Diagnostics  trainer,  which  closely 
approximated  the  actual  system.  Modeling  by  the  T.O.  caused  problems 
in  terms  of  keeping  the  tra'ners  up  to  date  as  flightline  procedures 
changed.  The  F- 16  maintenance  concept  is  procedure  oriented,  making 
modeling  by  T.O.  seem  reasonable.  However,  the  problems  caused  were 
severe,  as  the  simulation  was  often  incorrect.  This  raises  the  issue 
of  how  best  to  keep  a  trainer  current  with  the  operational  equipment 
being  used  on  the  flightline.  A  trainer  that  is  out  of  date  is 
annoying  to  the  instructors  and  confusing  to  the  students.  The  low 
fidelity  modeling  is  an  even  greater  problem  in  the  implementation  of 
automatic  student  monitoring,  as  will'  be  discussed  in  Section  4.2.2. 


4.2  Instructional  Features 


Two  instructiona  1  features  of  the  SAMTs  were  critiqued  by 
instructors:  malfunction  insertion  and  automatic  student  monitoring. 
Instructor  resoonses  concerning  these  two  features  indicated  that 
malfunction  insertion  was  generally  well  liked  and  that  student 
monitoring  was  universally  disliked.  The  possible  reasons  for  these 
differing  responses  are  discussed  below. 


4.2.1  Malfunction  Insertion  Feature 

The  responses  to  automatic  malfunction  insertion  were  extremely 
positive  in  three  out  of  the  four  courses.  Instructors  felt  that 
malfunction  isolation  could  be  taught  more  effectively  on  the  SAMTs 
than  on  AETs,  since  it  would  not  always  possible  to  create 
malfunctions  in  the  actual  equipment.  The  ability  to  show  students  the 
behavior  of  a  malfunctioning  system,  and  to  emphasize  the  results  of 
student  actions  in  a  realistic  context,  was  noted  as  one  of  the  best 
features  of  a  trainer.  It  is  worth  noting  that  the  evaluators  had 
difficulty  obtaining  comparative  data  on  the  feature,  because 
instructors  were  "sold"  on  the  importance  of  this  feature  and,  in  fact, 
were  unwilling  to  teach  without  it. 

However,  the  malfunction  feature  of  the  SAM^s  is  not  always 
perfect.  Instructors  noted  that  sometimes  malfunctions  were  simulated 
incorrectly,  or  procedures  programmed  into  the  trainer  were  out  of 
date.  (See  Section  4.1.4.) 

Some  comments  of  the  instructors  in  the  Avionics  course  were 
particularly  negative.  Their  main  criticism  did  not  seem  to  be  that 
the  malfunctions  were  badly  simulated,  but  that  the  appropriate 
troubleshooting  strategy  could  not  be  used  on  the  trainer.  That  is, 
the  trainer  did  not  allow  wiring  checks,  which  are  the  main  on-the-job 
troubleshooting  method.  Interestingly  (even  though  they  were  otherwise 
quite  negative  about  the  malfunction  feature)  the  Avionics  instructors 
stated  that  malfunction  capability  was  the  most  important  feature  of 
the  trainer.  Part  of  their  negative  response  seems  to  stem  from  a 
feeling  that  something  important  is  not  working  as  it  should.  The 
suggestion  here  is  that  trainers  must  be  designed  with  an  eye  to 
real-world  troubleshooting  methods,  as  well  as  realistic  modeling  of 
system  responses. 

It  should  also  be  noted  that  the  malfunction  exercises  are  pre¬ 
programmed.  Instructors  would  have  liked  to  be  able  to  change  these 
pre-programmed  exercises  to  make  them  more  realistic  (or  current  with 
flightline  procedures).  In  addition,  the  instructors  expressed  the 
desire  to  create  new  malfunction  exercises.  Currently,  the  SAMTs  do 
not  have  either  of  these  capabilities.  (See  also  Section  4.4.) 


4.2.2  Automatic  Student  Monitoring  Package 

The  automatic  student  monitoring  feature  is  part  of  an 
instructional  features  package  that  includes  automatic  freeze  and 
multiple  level  cueing.  Instructor  responses  were  obtained  for  all  the 
features  in  the  package. 


The  responses  to  this  package  were  overwhelmingly  negative.  The 
most  common  response  was  that  it  is  not  used  at  all.  A  number  of 
negative  points  seem  to  have  led  to  this  lack  of  utilization. 

One  point  is  that  the  monitoring  feature  is  "all  or  nothing";  that 
is,  when  monitoring  is  turned  on,  every  action  is  monitored.  At  the 
end  of  the  session,  all  of  these  data  are  reported  to  the  instructor. 
This  large  amount  of  information  is  difficult  to  read  and  interpret. 

The  instructors  would  have  preferred  to  be  able  to  select  the  actions 
to  be  monitored. 

Again,  the  modeling  of  the  student  monitoring  was  based  on  the 
T.O.  (See  Section  4.1.4.)  However,  the  instructional  features  were 
based  on  later  T.O.  versions  and  were  not  consistent  with  the  rest  of 
the  trainer.  Furthermore,  at  a  certain  point,  neither  the  main 
simulation  nor  the  student  monitoring  was  consistent  with  current 
technical  orders.  The  confusion  caused  by  this  is  easy  to  imagine. 
Several  T.O.  versions  had  to  be  kept  on  hand,  and  the  instructors  had 
to  try  and  explain  the  inconsistency. 

There  were  various  points  mentioned  about  error  messages.  The 
messages  specify  an  error  only  when  it  is  critical;  otherwise,  the 
message  states  only  that  an  error  was  made.  Also,  for  non-critical 
errors,  the  message  does  not  appear  until  the  end  of  t he  task.  This 
can  be  frustrating  for  the  student,  who  finds,  at  the  end  of  the  task, 
that  he/she  made  an  error,  but  does  not  know  what  it  was.  In  some 
cases,  since  the  system  models  out-of-date  technical  orders,  a  correct 
current  procedure  may  be  called  an  error  by  the  trainer.  Instructors 
have  the  capability  to  change  error  messages,  but  only  for  a  session, 
not  permanently.  This  means  that  the  same  incorrect  message  must  be 
changed  each  time  an  exercise  is  presented  to  the  student. 

Similar  problems  were  noted  with  the  automatic  freeze.  Sometimes, 
the  out-of-date  modeling  would  cause  a  freeze  on  a  correct  action. 
Instructors  would  have  liked  to  have  the  ability  to  change  error 
conditions,  both  tor  messages  and  for  simulator  freeze.  The  freeze 
feature  cannot  be  disabled,  so  instructors  cannot  shut  it  off  even  when 
they  know  it  will  react  incorrectly.  (See  Section  4.6  for  further 
discussion  of  the  freeze  feature.) 

The  cueing  feature  was  also  a  problem,  but  in  this  case,  it  seemed 
a  problem  of  instructor  training.  The  multi-level  cueing  was  evidently 
designed  to  be  used  with  a  successive  approximation  strategy.  That  is, 
at  the  beginning  of  training,  students  would  be  given  many  rues,  and  as 
training  progressed,  cues  would  be  reduced  until,  at  the  end  of 
training,  only  those  cues  available  on  the  job  would  be  seen.  The 
instructors'  response  to  cueing  was  that  it  was  not  realistic  because 


the  earlier  cues  would  not  be  available  on  the  job.  This  comment 
indicates  a  possible  lack  of  understanding  of  the  way  the  feature  is 
supposed  to  be  used.  Probably,  the  instructors  were  not  given  the  kind 
of  training  and  documentation  that  would  a1 low  them  to  use  this  feature 
effectively. 

Another  point  mentioned  about  the  cueing  feature  is  that  the 
messages  are  presented  on  the  instructor  S athod°-Ray  Tube  (CRT).  This 
configuration,  although  economical,  was  disliked  by  the  instructors. 

The  instructor  station  can  be  turnec  such  that  the  student  can  see  the 
CRT  while  at  the  panel,  but  then  tie  student  can  also  see  instructor 
commands  to  the  trainer.  A  sec  end  CRT  would  nave  been  beneficial. 

A  final  point  about  the  cueing  feat  ire  is  that  instructors 
mentioned  they  could  see  no  diffo- en~e  in  the  levels  of  cueing.  This 
may  be  a  fundamental  prob’e"1  '  n  r.os'y.  of  trie  cueing  feature  (i.e., 
the  cueing  messages  were  really  too  :  la-'  from  level  to  level),  or  it 
may  reflect  the  desire  of  fne  irt true  ter s  u.  be  able  to  change 
messages.  That  is,  poss'b’.v  ins  -,rs  just  aid  not  like  the 
particular  messages  that  -.-.ere  >ci. 

One  feature  that  was  1  <eJ  ;  chaining,  (i.e.,  the  ability 

to  preprogram  a  sequence  or  tasks'  7r  ,  feature  was  noted  as  saving 
time  and  improving  tine  f’ow  of  i  ■><.." «  <  ■-  i  i ...  . 

It  should  be  rioted  Lh,;f  'nst  'uc  dia  not  want  the  instructional 
features  package  to  begin  w.tn.  rney  t  ’It  that,  since  they  are  always 
in  the  room  to  observe  the  stucco.,  the  feature  was  not  needed.  This 
fact,  combined  with  a  lack  "  tra  io'-g  -.d  documentation,  and  the 
legitimate  problems  rioted  aoove,  as  er<.i-qh  to  ensure  the  feature  would 
not  be  used.  Th*  answer,  to  -  -re,  is  u  ^  o/ide  better  instructor 
training.  Also,  greater  control  over  error  conditions,  messages,  and 
monitored  actions  won lo  improve  : nstruc • acceptance.  (See  also 
Section  4.4.) 

A  final  point  wortn  mentioning  is  that  this  instructional  features 
package  increased  the  SnMTs'  need  for  computer  storage  to  such  an 
extent  that  a  costly  change  f roc  floppy  disks  to  hard  disks  was 
required.  Considering  also  the  expenses  associated  with  developing  the 
package,  it  seems  that  a  ;  -oncy  -<az,  perhaps,  not  wisely  spent, 
since  the  features  are  so  .-  >  <  •  I .  uses. 


4.3  Hardware  and  Sc-ftw^-e  i/'.-, 

Several  lessors  wer  .•  I  .  'nee  y-o  ‘  v  =  i r-er  hardware  and  software 
design.  Again,  these  lessor .  ?nt  ’'^present  u  complete  analysis  of 
trainers.  Rather,  these  or.-  net-'  s  r.  on  ernes gad  in  the  course  of  the 

study. 


.viv: 


?  v.  v.*.  u 


4.3.1  Component  InterchangeaPi  1 i  tv 

One  SAMT  feature  tnat  1-  considered  excellent  is  the  ability  to 
"swap"  components.  That  is,  the  coot; o1  coosole  and  components  of  the 
control  console  are  generally  'identical  frcm  SAMT  to  SAMT.  So,  if  one 
trainer  has  3  control  console  malfunction,  the  trainer  can  be  operated 
by  borrowing  another  consol*3,  or  by  replacing  a  bad  component  with  a 
borrowed  one,  until  a  replacement  is  obtained.  This  feature  is 
recommended  for  future  sets  of  trainers,  .it  was  noted  that 
occasionally  some  minor  configuration  differences  interfered  with  this 
practice. 

The  capability  to  exchange  master  :;;nsr.les  had  one  minor  drawback. 
Since  the  time  totalizer  for  the  tra;n=r  ,3-  a  whole  was  on  the  master 
console,  the  hours  of  use  of  each  trailer  were  not  clearly  recorded 
when  consoles  were  "swapped."  if  console  interchangeability  is 
specified,  it  is  recommended  that  separate  time  totalizers  for  consoles 
and  panels  be  specified  also. 


4.3.2  Computational  System  ? e jj a b j_ :  :y 

The  majority  of  maintenance  required  on  the  SAMTs  involved  the 
computer.  Maintenance  of  the  co'-’puc at  i  ;•  .a  i  system  is  a  difficult  area 
for  i nstructcrs/ma ’ *'t ai ners ,  and  in  the  case  of  the  SAMIS,  vendor 
assistance  is  often  req.j'reo  to  correct  :  .■mDute-'-re iated  malfunctions. 
This  is  costly  and  increases  oowrtii.ie,  vendor  personne1  must  be 

brought  in.  Also,  even  when  to*,  fault  might  be  repaired  by 
instructors/maintainers,  th-  ocr-utcr  parts  typically  needed  are  not 
readily  available.  «eliab  data  for  two  different  sites  were  very 
dissimilar,  with  Hill  AFC  -•  .;p<>-  t  i  no  f^.v  problems  with  the  SAMTs,  and 
McD ill  AFB  reporting  many.  Howev  ^  9  ;  '  i  <J  vJ  th  c res,  trie  computational 
system  was  noted  as  the-  cause-  cf  most  problems,  nlso,  instructors  at 
both  bases  stated  that  the  m v ufeoance  documentation  was  not 
sufficient.  (See  Table  T;  also  see  Sect  ior.s  3.3.5  and  5.3  for  further 
discussion  of  trainer  Maintenance. ; 

The  answer  to  the  problem  of  computer  reliablity  is  not  clear-cut. 
However,  at  the  very  least,  it  is  apparent  that  care  must  be  taken  in 
specifying  the  computational  syv^em  des’Ci.  Also,  the  "swapping" 
capability  described  ab  •:•*.:  :cj  Id  hc.c  a  ’  lace  recurring  computer 
faults,  by  permitting  one  tea  :,if«r  ‘  s  r  omeu'-  ?r  fc  be  used  for  another 
trainer  in  the  event  of  im!hin.;hj*v, 


4.3.3  Comput er  F e r ip her 3 1  c 

Several  of  the  peript'c-ral  u-;  c  • ..  e  the  SAMTs  seemed  to  be 
inadequate.  For  instance,  tie  fit  rev  T,*.  storage  proved  to  be  too 


Table  4.  Parts  for  SAMTs  at  McDill  AFB 


from  1  March  to  30  November  1981 

Component  Number  of  Replacements 

CPU  Boards  5 

GPI  Boards  10 

MDC  Boards  2 

Memory  Boards  27 

SIP  3 

CRT  3 

Line  Printers  2 

MAST  Components  2 

Disk  Drive  Components  1 

Battery  Back-up  2 

Lamp  Driver  4 

I/O  Control  1 

A/0  Converter 
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limited  when  instructional  features  were  added.  (See  Section  4.2.2.) 
It  was  also  mentioned  that  the  single  CRT  configuration  was  a  problem. 
(See  Section  4.2.2.)  Additionally,  it  was  noted  that  the  thermal 
printer,  while  adequate  for  limited  use,  would  have  been  a  problem  if 
use  was  expanded.  For  instance,  if  student  monitoring  was  used  more 
often,  or  if  the  printers  were  used  during  updates,  the  thermal  paper, 
which  is  difficult  to  fold  and  to  write  on,  would  have  created 
difficulties.  Because  the  instructors  dislike  the  printer,  they  have 
learned  how  to  "fool"  the  computer  so  as  to  disable  the  printer. 

The  suggestion  here  is  that  future  expansion  of  functions  should 
be  assumed  when  selecting  peripherals.  That  is,  specify  devices  which 
have  a  growth  capability. 


4.3.4  Use  of  Slide  Projector 

Another  useful  feature  of  the  SAMTs  had  to  do  with  the  slide 
projectors.  Each  SAMT  has  an  associated  slide  projector,  that  shows 
slides  of  components  upon  student  request  during  a  simulation  training 
session.  The  slide  projectors  could  also  be  operated  manually,  such 
that  they  could  be  used  independently  during  other  parts  of  the  course. 
This  type  of  flexibility  was  found  to  be  useful,  and  is  recommended 
whenever  devices  such  as  slide  projectors  are  to  be  utilized  with  a 
simulator.  Note  that  this  consideration  might  also  apply  to  film 
projectors,  videotape  players,  etc.,  depending  on  the  specific  trainer 
design. 


4.3.5  Trainer  Software 


The  SAMTs'  software  is  completely  transparent  to  the  student;  that 
is,  the  student  is  not  aware  of  how  the  simulation  works.  This  type  of 
design  is  considered  good.  The  student  does  not  require  much  training 
in  how  to  use  the  trainer  and  can  concentrate  on  the  tasks  to  be 
learned,  without  being  distracted  by  the  workings  of  the  training 
device. 

It  is  not  known  whether  the  SAMT  software  is  designed  in  such  a 
way  that  it  could  be  reused.  That  is,  it  is  possible  that  some  of  the 
simulation  or  instructional  features  software  could  be  used  in  future 
trainers.  If  so,  the  cost  of  developing  software  will  be  less  in 
future  acquisitions. 


4.4  Update  Capability 

The  SAMTs  have  very  little  capability  for  update  by  the  instruc¬ 
tors  (or  instructors/maintainers).  Especially  because  of  the  modeling 
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problems,  the  simulation  was  often  out  of  date.  (See  Sections  4.1.4 
and  4.2.2.)  The  capability  to  perform  on-site  updates,  rather  than 
having  to  wait  for  vendor  modifications,  would  have  saved  time  and 
money.  Even  a  fairly  limited  capacity  for  Instructors  to  define  or 
modify  exercises  would  have  increased  the  instructors'  ability  to  work 
around  incorrect  simulation.  This  need  for  greater  instructor  control 
of  simulation  seems  to  be  a  recurring  theme.  (See  Section  4.2.2.) 

8oth  instructor  acceptance  and  instructional  value  would  be  increased 
with  greater  instructional  control. 

It  should  be  noted  that  the  capability  to  select  values  or  alter 
parameters  is  not  the  same  as  the  need  to  do  so.  Instructors  should 
not  have  to  worry  about  the  simulation  all  the  time,  but  should  at 
least  be  able  to  change  those  points  which  require  changing.  This 
suggests,  for  instance,  that  default  values  for  selectable  parameters 
should  always  be  provided. 


4.5  Design  for  the  Training  Context 

A  couple  of  points  were  noted  about  the  tasks  simulated  and  the 
general  approach  to  training.  The  trainers'  design  does  not  seem  to 
have  been  perfectly  matched  to  the  training  situation;  two  examples  are 
provided  in  the  next  two  paragraphs.  (See  Sections  4.5.1  and  4.5.2.) 


4.5.1  Choice  of  Equipment  to  Simulate 

In  studying  student  task  performance,  it  was  noted  that  certain 
tasks  did  not  seem  to  have  been  taught  at  all.  For  instance,  students 
were  completely  unfamiliar  with  two  different  test  sets  included  in  two 
of  the  SAMTs.  Somehow,  the  decision  had  been  made  to  include  those 
test  sets  in  the  trainers,  despite  the  fact  that  the  instructors  do 
not  actually  use  them.  The  problem  seems  to  be  that  instructors  do  not 
necessarily  teach  all  tasks  listed  on  a  course  syllabus.  It  appears 
that  the  instructors  believe  that  only  certain  tasks  require  practice. 
The  tasks  believed  to  require  practice  will  be  taught  using  the 
simulator  and  others  will  not. 

To  avoid  including  items  in  the  trainer  that  will  not  be  used, 
greater  instructor  involvement  in  the  design  process  is  needed.  It  is 
known  that  at  least  one  instructor  (in  each  system)  was  used  as  a 
Subject  Matter  Expert  in  the  SAMT  design  process.  It  might  be  better 
to  have  several  instructors  from  each  field,  perhaps  with  one  as  the 
main  consultant. 


4.5.2.  Student  Level 


When  the  SAMTs  were  initially  installed,  they  were  used  primarily 
for  conversion  training  (that  Is,  taking  personnel  qualified  on  one 
aircraft  and  training  them  or  exposing  them  to  the  aircraft  of  inter¬ 
est).  Later,  the  primary  use  was  training  3-levels  directly  out  of 
technical  school.  It  was  mentioned  to  the  evaluators,  in  conversations 
with  SPO  personnel,  that  the  SAMTs  had  been  basically  designed  with 
conversion  training  in  mind,  since  that  was  to  be  their  first  use.  In 
general,  conversion  training  will  occur  during  the  first  3  or  4  years 
in  the  life  cycle  of  a  weapon  system.  Also,  it  has  been  suggested  that 
3- levels  may  have  more  need  for  hands-on  practice  than  do  higher  level 
technicians.  It  would  have  probably  been  better  to  design  for  the 
lowest  level  student  expected  to  use  the  trainer  over  its  life  cycle. 
This  is  not  to  suggest  that  higher  level  students  do  not  benefit  from 
simulator  training,  just  that  3-levels  get  the  most  benefit  from  it 
and,  during  the  life  of  the  trainer,  will  represent  the  largest  group 
of  students;  therefore,  it  should  be  designed  to  that  level. 


4.6  Trainer  Operation 

In  general,  the  Instructors  had  few  complaints  about  trainer 
operation  procedures.  However,  it  was  noted  that  the  initialization 
procedures  were  too  complex.  The  system  had  to  be  "booted  up"  before 
exercises  could  be  loaded,  which  required  that  two  separate  floppy 
disks  be  inserted.  The  initialization  procedures  took  between  5  and  10 
minutes  to  complete.  This  was  a  problem  because  the  system  had  to  be 
reinitialized  after  a  freeze.  Initialization  procedures  should  be  as 
brief  as  possible,  and  the  freeze  capability  should  not  require 
reinitialization. 
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Section  5:  Trainer  Evaluation  Issues 


The  connection  between  evaluating  trainers  and  acquiring  trainers 
is  sometimes  forgotten.  It  is  necessary  to  evaluate  past  trainers  to 
know  which  features  are  good  and  which  are  not,  so  that  the  good  ones 
can  be  acquired  again,  and  the  bad  ones  avoided.  In  other  words, 
trainer  evaluations  result  in  an  improved  capability  to  acquire  good 
training  devices.  This  section  presents  lessons  learned  about 
conducting  trainer  evaluations. 


5. 1  Cost  Reporting 

The  evaluators  had  a  great  deal  of  difficulty  obtaining  cost  data, 
for  a  number  of  reasons.  (See  Section  5.2.2.)  One  of  the  main 
factors,  however,  was  the  fact  that  cost  data  were  simply  not  reported 
to  the  Air  Force  in  a  usable  form.  That  is,  the  data  were  not  reported 
in  a  form  suitable  for  use  in  a  cost  analysis.  For  instance,  there  was 
no  way  to  determine  how  much  of  the  SAMTs'  costs  were  related  to  the 
instructional  features. 

It  appears  that  this  gross  level  of  cost  reporting  is  standard  in 
the  military  environment.  The  planned  use  of  the  data  is  to  monitor 
acquisition  costs,  so  more  detailed  data  are  not  required  from  contrac¬ 
tors.  This  has  several  implications  for  conducting  cost  studies.  For 
instance,  the  use  of  standard  forms  will  probably  not  be  adequate. 

That  is,  military  personnel  involved  in  the  study  will  have  to  be 
willing  to  record  additional  information  for  the  study,  not  just  the 
data  they  always  record.  (For  further  discussion  on  the  use  of 
military  personnel  in  evaluations,  see  Section  5.2.) 

It  is  conceivable  that  a  cost  model  could  be  developed  that  would 
make  optimal  use  of  the  available  data.  However,  only  so  much  can  be 
done  without  adequate  information.  Because  of  this,  it  is  probably 
necessary  to  change  Air  Force  cost  reporting  procedures,  if  effective 
cost  studies  are  to  be  performed.  Cost  reporting  procedures  must  be 
developed  that  will  allow  useful  data  to  be  extracted,  and  contractors 
must  be  required  to  use  these  procedures.  Standardizing  the  procedures 
will  allow  comparisons  to  be  made  easily  and  directly. 


5.2  Data  Collection 


Collecting  data  in  the  military  training  environment  always 
presents  certain  difficulties.  The  experiences  of  the  F- 16  SAWT  study 
provide  information  about  what  problems  can  occur  and  how  to  get  around 
them. 
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5.2.1  Training  Effectiveness  Data 


A  number  of  small  problems,  and  one  major  difficulty,  were  asso¬ 
ciated  with  the  collection  of  training  data.  These  problems  emerged 
even  though  the  study  was  carefully  planned,  research  agreements  were 
made  early,  and  a  pilot  study  was  done  to  identify  any  potential 
problems.  Unfortunately,  as  the  study  continued,  problems  emerged 
which  were  thought  to  have  been  previously  resolved  or  accounted  for. 
Each  of  these  is  discussed  below. 

Several  problems  caused  delays  in  the  collection  of  training 
effectiveness  data.  Data  collection  took  about  twice  as  long  as 
originally  planned.  An  example  of  the  type  of  problem  encountered  was 
that  initial  data  on  student  flow  were  incorrect.  The  data 
overestimated  the  proportion  of  eligible  3-levels  in  the  classes  and 
did  not  take  Into  account  the  large  number  of  foreign  students  who  were 
trained  at  Hill  AFB  during  the  evaluation.  Since  it  was  desirable  to 
have  all  students  with  a  similar  background,  this  faulty  information 
resulted  in  incorrect  planning.  It  took  much  longer  than  expected  to 
get  enough  students  (even  when  5-levels  were  included). 

Another  example  is  that  the  evaluators  could  not  obtain  working 
test  sets  for  hands-on  evaluations,  even  though  this  was  a  condition 
ipeclfled  in  the  research  agreement.  Reasonably  enough,  the  priority 
was  for  test  sets  to  be  used  in  maintenance.  However,  the 
unavailability  of  test  sets  for  use  in  the  evaluation  caused  many 
delays.  Again,  initial  Information  had  indicated  that  the  test  sets 
would  be  available. 

Oelays  tended  to  snowball,  and  to  cause  other  unforeseen  problems. 
For  Instance,  because  the  evaluation  took  longer  than  expected,  some 
data  collection  forms  had  to  be  updated;  this  caused  more  delay.  Also, 
students  were  sometimes  transferred  off  base  shortly  after  training,  so 
if  their  task  performance  was  not  observed  immediately  (because  a  test 
set  was  not  available),  they  would  be  gone  and  the  data  could  not  be 
col lected. 

These  experiences  suggest  that  time  estimates  for  data  collection 
must  be  made  with  extreme  caution.  Probably,  the  best  initial  estimate 
should  be  doubled  to  allow  for  unforeseen  problems. 

The  major  problem,  however,  was  limited  resources.  The  data 
collection  efforts  Involved  personnel  at  FTD,  Tactical  Air  Command 
(TAC),  and  several  Aircraft  Maintenance  Units  (AMUs).  The  many 
different  lines  of  authority  involved  made  communication  and  control 
very  difficult.  Although  cooperation  was  usually  excellent,  problems 
occasionally  occurred.  Air  Force  personnel  not  directly  under  the 
authority  of  evaluation  contacts  tended  to  be  less  cooperative. 


Presumably,  personnel  with  responsibilities  of  their  own  may  consider 
an  evaluation  of  a  device  they  do  not  even  work  with  to  be  a  low 
priority. 

Although  FTO  instructors  were  extremely  helpful,  they  were 
reluctant  to  change  their  teaching  methods  for  evaluation  purposes. 

For  instance,  they  did  not  want  to  teach  without  the  malfunction 
capability  because  they  believed  it  was  useful  and  important.  However, 
this  made  it  nearly  impossible  to  obtain  comparative  data  on  the 
effectiveness  of  the  feature. 

The  F- 16  SAMT  study  had  extremely  limited  resources  and  could  not 
have  personnel  on-site  full  time.  The  experiences  described  above 
suggest  that  evaluations  cannot  be  successfully  done  under  those 
conditions.  (See  also  Section  5.2.2.)  Much  greater  resources  would 
have  been  needed  even  to  monitor  how  instructors  were  teaching,  much 
less  enforce  evaluation  guidelines.  Furthermore,  in  some  cases, 
conducting  evaluations  in  the  actual  training  environment  simply  may 
not  be  feasible.  That  is,  in  some  cases,  it  may  be  necessary  to  set  up 
an  artificial  situation,  rather  than  using  actual  students  in  actual 
training  courses.  This,  too,  would  require  that  greater  resources  be 
devoted  to  a  study. 


5.2.2  Cost  Effectiveness  Data 

Obtaining  usable  cost  data  can  be  extremely  difficult.  Military 
guidelines  make  it  nearly  impossible  for  one  contractor  to  review  cost 
figures  from  another  (even  if  the  two  contractors  are  not  in 
competition).  Often,  a  contractor  cannot  obtain  the  release  of 
sensitive  information.  Also,  data  may  contain  hidden  variability.  For 
instance,  the  downtime  figures  reported  to  the  F- 16  SAMT  study  did  not 
allow  for  console  swapping.  (See  Section  4.3.1.) 

These  problems  suggest  a  couple  of  different  remedies.  First, 
high  level  military  personnel  should  be  involved  in  the  cost  study, 
rather  than  just  contractor  personnel.  Also,  someone  should  be  on  site 
full  time  to  track  down  cost  data.  Air  Force  accounting  procedures 
make  it  necessary  to  trace  information  through  many  individuals  and 
branches,  to  get  the  whole  picture.  In  the  F- 16  SAMT  study,  the 
resources  to  do  this  were  not  available.  This  suggests  that  Air  Force 
accounting  procedures  and/or  procedures  used  by  vendors  to  report  cost 
data  to  the  Government  should  be  studied.  The  data  reported  by  vendors 
should  be  in  a  format  that  is  usable  in  a  detailed  cost  analysis. 


Section  6:  Training  System  Issues 


A  training  device  cannot  be  designed  in  isolation  from  the 
training  context.  The  lessons  that  are  presented  in  this  section  have 
to  do  with  the  F- 16  training  program,  and  with  Air  Force  maintenance 
training  in  general. 


6.1  Procedural  Approach 


The  F-16  maintenance  concept  emphasizes  procedure-following, 
rather  than  a  system  logic  approach.  In  observing  student  task 
performance,  it  was  found  that  students  often  had  little  understanding 
of  system  logic.  Frequently,  they  could  not  explain  task  steps.  That 
is,  when  they  were  following  a  procedure,  they  often  did  not  know  what 
the  steps  were  for  or  how  they  related  to  the  system  being  checked. 


This  lack  of  understanding  of  system  logic  may  not  be  a  major 
problem.  Since  the  maintenance  concept  emphasizes  procedures,  it  can 
be  argued  that  technicians  do  not  need  to  understand  the  underlying 
theory  of  what  they  are  doing.  However,  there  is  a  point  where  system 
knowledge  becomes  essential.  Procedures  cannot  cover  every  possible 
fault  in  a  complex  system.  It  should  be  noted  that  instructors  believe 
that  system  logic  is  important  and  try  to  stress  it;  however,  the  F-16 
courses  are  not  designed  with  this  emphasis.  Although  the  issue  is 
open  to  question,  the  evidence  does  suggest  a  lack  in  the  training  of 
maintenance  technicians.  A  strong  empahsis  on  system  logic  would  be 
expected  to  produce  better  technicians,  who  can  troubleshoot  systems 
more  effectively  and  efficiently. 


6.2  Aircraft  Time 


Students  in  the  F-16  courses  stated  that  they  would  like  to  have 
spent  more  time  working  on  the  actual  aircraft.  Although  they  liked 
the  trainers  and  did  not  want  less  time  on  the  SAMTs,  they  also  felt 
that  they  learned  more  in  OJT  working  on  the  aircraft  than  they  had  in 
the  SAMT  courses.  Observations  of  task  performance  also  suggested  that 
greater  exposure  to  the  aircraft  might  have  been  helpful.  (See  Section 
4.1.2.) 


Although  the  suggestion  seems  to  be  that  these  particular  simula¬ 
tors  add  to  training,  they  cannot  entirely  replace  experience  on  and 
exposure  to  the  actual  aircraft.  Courses  utilizing  trainers  should  be 
designed  with  this  in  mind. 

In  the  case  of  F-16  training,  OJT  is  supposed  to  provide  actual 
aircraft  exposure  and  practice.  However,  quite  often  the  3-levels  in 
OJT  performed  few  of  the  tasks  they  were  supposed  to  be  learning.  More 
experienced  technicians  tended  to  do  the  complex  maintenance  tasks, 


31 


while  the  OJT  students  swept  the  hangar  or  cleaned  up  spilled  hydraulic 
fluid.  Obviously,  most  students  were  learning  something  and  some 
students  got  a  lot  of  practice  at  skills  they  needed  to  acquire. 
However,  their  utilization  varied  widely  from  unit  to  unit,  and  some 
students  got  very  little  useful  practice.  In  order  for  OJT  to  be  an 
effective  and  consistent  learning  experience,  perhaps  more  structure  is 
needed. 


6.3  Trainer  Maintenance  Job 

The  SAMTs  are  maintained  by  instructors,  with  occasional  vendor 
assistance.  This  works  fairly  well,  because  the  Electrical  and 
Avionics  instructors,  as  Subject  Matter  Experts  in  their  fields,  have 
skills  that  can  be  transferred  to  maintaining  the  trainers.  In  cases 
where  instructors  do  have  applicable  knowledge,  it  makes  sense  for  them 
to  be  responsible  for  trainer  maintenance. 

However,  the  maintenance  of  the  trainers  is  essentially  done  in 
the  instructors'  spare  time.  They  are  scheduled  to  teach  6  hours  a 
day,  with  2  hours  for  preparation.  This  does  not  leave  time  for 
trainer  maintenance. 

It  is  difficult  to  tell  exactly  how  burdensome  this  is.  Data  from 
two  different  sites  varied  greatly,  as  shown  in  Table  5.  However,  even 
if  only  a  few  hours  a  month  are  spent  on  maintenance,  it  seems  unfair 
for  those  hours  to  come  from  instructors'  free  time. 


The  alternative  is  to  set  up  a  separate  job  position  for  the 
maintainer.  The  basic  idea  of  using  instructors  as  maintainers  seems 
to  work.  An  instructor  could  be  put  in  charge  of  both  routine 
maintenance  and  major  courseware  changes.  Having  an  instructor  in  this 
position  would  assure  that  the  concerns  of  the  devices'  users  were 
addressed.  In  addition,  having  the  instructor/maintainer  freed  from 
teaching  duties  would  assure  sufficient  attention  was  given  to  device 
management.  Note  that  the  choice  of  having  one  person  serve  as  a  full¬ 
time  maintainer  is  arbitrary.  Depending  on  maintenance  requirements, 
more  or  less  time  would  be  needed.  The  point  is  that  maintenance  of 
the  trainer  should  not  be  done  in  someone's  spare  time. 
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Section  7:  Conclusion 


The  original  purpose  of  this  study  was  not  only  to  collect  data  on 
the  SAMTS,  but  also  to  gather  information  to  assist  future  design  and 
research  efforts.  The  SAMT  evaluation  did  indeed  learn  much  that  is 
pertinent  to  trainer  acquisition  planning  as  well  as  trainer 
evaluation.  Many  aspects  of  the  study  were  difficult  to  conduct;  but 
often  the  difficulties  themeselves  were  informative.  It  is  hoped  that 
the  “lessons  learned"  presented  in  this  document  will  be  of  interest 
and  assistance. 
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INOEX 


Actual  Equipment  Trainers  (AET).  3.0,  3.3,  3.4 

Aircraft,  use  in  training  3.3.2,  3.3.5,  3.4,  4.1.2,  6.2 

Automatic  freeze  4.2.2,  4.6 

Automatic  student  monitoring  package  3.0,  4.2.2 

Component  interchangeability  4.3.1,  5.2.2 

Computer 

Maintenance  3.3.5,  4.3.2,  6.3 
Peripheral  devices  4.2.2,  4.3.3 
Reliability  3.3.5,  4.3.2 
Storage  4.2.2,  4.3.3 

Conversion  training  2.0,  4.5.2 

Cost  data  3.0,  3.2.2,  3.3.1,  3.3.3,  5.1,  5.2.2 

Cost  effectiveness  3.0,  4.2.2,  4.3.1,  4.3.2,  4.3.5,  4.4,  5.1,  5.2.2 

Cost  factors,  see  Costs 

Cost  model  3.1,  5.1 

Costs 

Acquisition  3.2,  3.3.4 
AET  3.0,  3.3,  3.4 

Computer  3.3.5,  3.3.6,  4.2.2,  4.3.1,  4.3.2,  4.3.5 

Facility  3.3.1 

Fuel  3.3.1,  3.3.5 

Instructional  features  3.0,  4.2.2 

Instructor  salary  3.3.1,  3.3.5,  6.3 

Maintenance  3.3.1,  3.3.5,  4.3.1,  4.3.2,  5.1 

Modification  and  update  3.3.6,  4.4 

Operation  3.3.1,  3.3.5 

CRT  4.2.2,  4.2.3 

Cueing,  automatic  4.2.1 


INOEX  (continued) 


Disks  4.2.2,  4.3.3,  4.6 

Emergency  procedures  3.4,  4.1.1,  4.2.1 

Exercise  creation  4.2.1,  4.4 

F-16  maintenance  concept  6.1 

Fidelity  levels  2.0,  4.1,  4.2 

Floppy  disks,  see  Disks 

Freeze  capability,  see  Automatic  freeze 

Hard  disks,  see  Disks 

Initialization  4.6 

Instructional  features,  or  instructional 
support  features,  see  Automatic 
student  monitoring  package,  Malfunction 
simulation  capability 

Instructional  value  3.4,  4.0,  6.2 

Instructor  attitude  2.0,  4.2,  4.3.3,  4.4,  4.5.1,  4.6,  5.2.1 

Instructor  station,  see  Master  simulation 
control  console 

Instructor/maintainer  job  3.3.5,  4.3.2,  4.4,  6.3 

Lesson  chaining  4.2.1,  4.2.2 

Local  changes,  see  Update 

Logistical  Support,  see  Costs,  Maintenance 
and  Operation 

Maintenance  by  instructors,  see  Instructor/ 
maintainer  job 

Maintenance  of  trainer,  see  Trainer  maintenance 
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INDEX  (continued) 


Malfunction  simulation  capability  (malfunction 
insertion)  3.4,  4.2.1,  5.2.1 

Master  Simulation  Control  Console  (MSCC)  2.0,  4.2.2,  4.3.1,  4.6 

Modeling  4.1.4,  4.2.2 

Modification,  see  Update  _  ...  . 

Performance  monitoring,  see  Automatic  student 
monitoring  package 

Printer  4.3.3 

Procedural  troubleshooting  4.1.1,  4.1.4,  6.1 
Simulation  system,  see  Software 
Slide  projector  2.0,  4.1.2,  4.3.4 
Software  3.3.5,  3.3.6,  4.1.4,  4.3.5,  4.4 
Student  attitude  4.1.1,  6.2 

Student  skill  level  2.0,  4.1.1,  4.1.2,  4.1.3,  4.5.2,  6.1 
System  mathematical  modeling,  see  Modeling 

Trainer  operation  3.3.1,  3.3.5,  4.2.1,  4.2.2,  4.2.3,  4.3.4,  4.3.5, 
4.5.1,  4.6,  5.2.1 

Trainer  maintenance  3.3.1,  3.3.5,  4.3.1,  4.3.2,  6.3 
Update  3.3.6,  4.1.4,  4.2.1,  4.2.2,  4.3.3,  4.4,  6.3 
Vendor  maintenance  3.3.1,  3.3.5,  4.3.2 
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